An Approximate L1-Difference Algorithm for Massive Data Streams

We give a space-efficient, one-pass algorithm for approximating the L1 difference Pi jai bij between two functions, when the function values ai and bi are given as data streams, and their order is chosen by an adversary. Our main technical innovation is a method of constructing families fVjg of limited-independence random variables that are range-summable, by which we mean that Pc 1 j=0 Vj(s) is computable in time polylog(c), for all seeds s. These random-variable families may be of interest outside our current application domain, i.e., massive data streams generated by communication networks. Our L1-difference algorithm can be viewed as a “sketching” algorithm, in the sense of [Broder, Charikar, Frieze, and Mitzenmacher, STOC ’98, pp. 327-336], and our algorithm performs better than that of Broder et al. when used to approximate the symmetric difference of two sets with small symmetric difference.